Force feedback computer input and output device with coordinated haptic elements

ABSTRACT

A set of haptic elements (haptels) are arranged in a grid. Each haptel is a haptic feedback device with linear motion and a touchable surface substantially perpendicular to the direction of motion. In a preferred embodiment, each haptel has a position sensor which measures the vertical position of the surface within its range of travel, a linear actuator which provides a controllable vertical bi-directional feedback force, and a touch location sensor on the touchable surface. All haptels have their sensors and effectors interfaced to a control processor. The touch location sensor readings are processed and sent to a computer, which returns the type of haptic response to use for each touch in progress. The control processor reads the position sensors, derives velocity, acceleration, net force and applied force measurements, and computes the desired force response for each haptel. The haptels are coordinated such that force feedback for a single touch is distributed across all haptels involved. This enables the feel of the haptic response to be independent of where touch is located and how many haptels are involved in the touch. As a touch moves across the device, haptels are added and removed from the coordination set such that the user experiences an uninterrupted haptic effect. Because the touch surface is comprised of a multiple haptels, the device can provide multiple simultaneous interactions, limited only by the size of the surface and the number of haptels. The size of the haptels determines the minimum distance between independent touches on the surface, but otherwise does not affect the properties of the device. Thus, the device is a pointing device for graphical user interfaces which provides dynamic haptic feedback under application control for multiple simultaneous interactions.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims benefit of priorityunder U.S. patent application Ser. No. 09/357,727 filed Jul. 21, 1999,now U.S. Pat. No. 6,337,678 and which is incorporated herein byreference in its entirety.

BACKGROUND

1. Field of Invention

This invention relates to computer input and output devices,specifically to those which provide force feedback, and to those whichcan be used as a pointing device for graphical user interfaces.

2. Description of Prior Art

Computers are becoming increasingly important as a productivity tool.They continue to improve dramatically in terms of computational speed,memory, storage and display. However, the interface between users andthe computer has not changed significantly since the introduction of themouse and the graphical user interface. The human-computer interfacemust be improved for users to increase their productivity and takebetter advantage of the new capabilities computers provide.

Many common computer interface operations are best performed with adirect manipulation interface. For example, when using a drawingapplication, it is easier for the user to point at the object they wishto select, rather than use a voice recognition interface in which theymust describe the object they wish to select. Typically, directmanipulation interfaces combine a high-resolution pointing device, usedto move a cursor on the screen, with some way to initiate an action atthe current location. For example, a mouse may employ rotary opticalencoders to measure the distance moved, and one or more buttons for“clicking” on the object beneath the cursor (e.g., selecting, actuating,dragging, or otherwise manipulating an on-screen object.).

While this was a significant improvement over previous devices, such aninterface does not come close to fully exploiting the abilities peoplehave to manipulate objects with their hands. Existing devices have oneor more of the following drawbacks:

No Direct Mapping Between the Hand and the Display

Direct mapping is used herein to describe the case where a one-to-onecorrespondence exists between the position of a cursor on a screen andthe position of a user's hand, and also implies that there is a uniquehand position for every cursor position. Input devices which do notmove, such as trackballs, joysticks, the IBM TrackPoint™ and theSynaptics TouchPad, lack such a direct mapping. No matter where thecursor is, the user's hand is in essentially the same location. A mousealso lacks a direct mapping, for at least two reasons. First, there is anon-linear relationship between the speed of the mouse and the speed ofthe cursor on the screen. This results in a different position dependingon how quickly the mouse is moved from one location to another. Second,the mouse is often picked up and moved during use, particularly if theworking area is limited.

Direct mapping is important because it better leverages a user's spatialskills. Humans have a keen sense of the position of their hands inrelationship to their body and their environment. Taking advantage ofthese spatial skills is valuable because the cognitive load placed onthe user by the computer interface is decreased, leaving the user'sattention available for performing work. For example, when dragging anobject from one point on the screen to another, a user must pay closeattention to a cursor's position and look for visual feedback indicatingthe cursor is positioned properly, in order to manipulate an on-screenobject. During this process, the user's attention is not available forother tasks (e.g., reviewing files, program output, and the like). Someexisting input devices have a direct mapping between the hand and thescreen, such as touch screens and digitizing tablets. These devicessuffer from other infirmities, as described below.

Lack of Dynamic Haptic Feedback

Haptic feedback is a preferable characteristic for input devices. Theterm haptic feedback as used herein means communicating information to auser through forces applied to the user's body. Typically, the positionof some portion of an input device changes along at least one degree offreedom depending on the force applied by the user. For example, whenpressing a button on a mouse, the button does not move until the appliedforce reaches a certain threshold, at which point the button movesdownward with relative ease and then stops (e.g., the sensation of“clicking” a button). The change in the position of the buttoncommunicates to the user through their sense of touch that the mouseclick was successful. Note that a device with haptic feedback can be aninput device (initiating an action) and an output device (giving hapticfeedback indicating that the action was initiated) simultaneously.

Input devices that are completely devoid of haptic feedback, such asmembrane keyboards and touch screens, have not gained widespreadacceptance for desktop computers as a result of this deficiency. Thuswhen using such input devices, users are uncertain whether a fingerpress was registered by the computer and so must pay special attentionto visual or auditory feedback to get this confirmation. This decreasesdata entry rates, making users less productive and the computerinterface less enjoyable to use.

Mice, trackballs, joysticks, and other devices often provide buttons forinitiating actions that provide haptic feedback. For example, the stylusused with a graphics tablet has a spring in its tip so the position ofthe pen relative to the tablet can vary depending on the applied force.However, such devices have the same haptic response regardless of thestate of the user interface. For example, if a user clicks the mouse ona graphical button that is disabled, the haptic response of the mousebutton is no different from that of clicking a button that is enabled,and so is misleading to the user because no action will result from theclick. What is needed is an input device which provides dynamic hapticfeedback. Haptic feedback is termed herein as being dynamic to indicatethat the haptic feedback can be altered over time (e.g. by means asoftware application) in order to provide additional information to auser.

A number of devices having dynamic force feedback exist. Most of theselack a direct mapping between the hand and the device (e.g.force-feedback joysticks). Others have a direct mapping but areprimarily designed for use in three-dimensional applications such asvirtual reality or tele-operation. Most productive work done oncomputers is two-dimensional in nature, such as spreadsheets and pagelayout. These productivity applications would not enjoy significantbenefits from the use of a three-dimensional input device. These deviceshave additional drawbacks, as outlined below.

User Interaction is Encumbered or Impeded

Many input devices encumber the user by requiring them to move at leasta portion of the input device during use. For example, the time it takesto move the cursor across the screen with a mouse is increased becausethe user must accelerate and decelerate the mass of the mouse, inaddition to the mass of their hand. Other input devices do not addinertia but impede the user in other ways. With a trackball, forexample, multiple sweeping motions are required to move the cursor largedistances, which is awkward and time consuming. With a joystick, forexample, the force applied relates to the speed of the cursor on thescreen, which may require the user to wait when the cursor is movingrelatively large distances.

Any input device which must be located and/or manipulated before usesuffers from such problems to at least a certain extent (e.g., mice andsome force reflecting interfaces, among others). For example, if aperson not currently using a computer and wants to press a graphicalbutton on computer's display, they must find and grasp the mouse, movethe mouse to position the cursor over the button, and then click thebutton. In contrast, a touch screen leaves the user unencumbered. Theycan reach out and press a graphical button on the display directly, withno intermediate steps. A touch screen, however, suffers from thepreviously-described infirmity of lacking haptic feedback.

Insufficient Support for Multiple Interactions

Most input devices, such as the mouse, trackball, joystick, theSynaptics TouchPad and the IBM TrackPoint™, only support a singleinteraction at a time. However, people have two hands which they areinnately able to use together. Two single-interaction devices have beencombined to provide two points of control, but confusion can arisebecause the correspondence between screen cursors and pointing devicesis not apparent. Because these devices lack a direct mapping to thescreen, their physical positions cannot resolve the correspondencebetween an input device and its cursor. Moreover, no provision is madefor the interaction of multiple users. With a single input device, onlya single user may “own” the device at any given time, and (given asingle input device) users must take turns interacting with thecomputer. This is obviously a cumbersome and awkward technique whenmultiple users wish to work collaboratively on a given project.

SUMMARY OF THE INVENTION

Embodiments of the present invention overcomes conventional limitationsby providing a device having a direct mapping, for example, between thetouching portion of a user's hand and the position of a cursor on adisplay and an output in the form of dynamic haptic feedback, withoutencumbering or impeding the user and allowing a large number ofsimultaneous interactions. The device provides direct mapping to reducethe conscious effort required for relatively pedestrian tasks such asinteracting with a graphical user interface (GUI). The user'sinteraction with the device is not hampered by a need to laterally moveany portion of the device.

The device provides dynamic haptic feedback. Haptic feedback is termedherein as being dynamic to indicate that the haptic feedback can bealtered over time (e.g. by means a software application) in order toprovide additional information to a user. In the previous example, adisabled button would have a different feel from that of an enabledbutton, allowing a user to discern that a graphical button was notenabled, using their sense of touch. The device also supports multipleinteractions. Having more than two points of control is useful whenmultiple users collaborate at the same computer. Allowing a large numberof interactions at once allows multiple users to interact with thecomputer simultaneously. Another benefit of having more than two pointsof control is the ability of a user to employ multiple fingers forpointing purposes, even in combination.

Embodiments of the present invention take the form of an input andoutput device for a processor. In one embodiment, an input/output devicehas a horizontal two-dimensional area which can be touchedsimultaneously (e.g., with the hands) in multiple places. The locationof each touch is measured and the area surrounding each touch movesvertically and provides dynamic haptic feedback to the user. The devicehas a control processor that communicates with another processor onwhich software applications are executed. The control processorcontinually sends the current attributes of all touches in progress, andreceives commands which specify the type of haptic response each touchshould exhibit.

The touchable area is comprised of a grid of haptic elements, referredto herein as haptels. Haptel is used herein to describe a hapticfeedback device with linear motion having a touchable surfacesubstantially perpendicular to the direction of motion. A hapticfeedback device is used herein to describe an input and output devicewith a moving portion manipulated by a user, one or more sensors thatmeasure the position and/or various derivatives of position and/or theforces applied to the moving portion, one or more effectors which canapply forces to the moving portion, and a processor which measures thesensors, computes a response, and drives the effectors to create a rangeof haptic effects.

In one embodiment, each haptel includes a position sensor to measure thevertical position of the surface within its range of travel, anelectromagnetic linear actuator to provide a controllable verticalbi-directional feedback force, and a touch location sensor to measurethe coordinates of a single touch within its bounds. Preferably, thehaptel grid is covered by a single sheet of flexible material thatprotects the haptels and hides the grid from view.

The haptels have their sensors and effectors interfaced to a controlprocessor. The control processor measures the position of haptelsurfaces and allows information such as velocity, acceleration, andapplied force to be derived. Alternatively, sensors can be included ineach haptel to provide such measurements (and others) directly. Thecontrol processor computes the desired feedback force for each hapteland drives the actuators to generate the appropriate forces. The hapticresponse of each haptel may be configured to be essentially arbitrarywithin a certain range. The range of available effects depends on thetype of sensors employed, the bandwidth and precision of the sensors andeffectors, the resolution of the analog-to-digital and digital-to-analogconversion performed, the amount of available processing power and theupdate frequency of the control loop, among other factors. Thesetradeoffs would be apparent to one skilled in the art of force feedbackdesign.

Because the touchable area is comprised of many haptels, each of whichcan function independently, the device allows multiple touches at once.Each haptel responds to only one touch at a time, so that there is alower bound on the distance between two touches which do not interferewith each other. The worst-case value of this minimum distance isapproximately the diagonal size of a haptel. However, in a specificinstance the minimum distance can be substantially smaller depending onthe locations of the two touches. Smaller haptels allow touches to becloser to one another.

A typical interaction is a user pressing a graphical button displayed aspart of a GUI. The finger touches the device, landing on a specifichaptel. The overall location of the touch is determined by the touchlocation sensor of the haptel in combination with the location of thathaptel within the haptel grid. The touch location is communicated to aprocessor (e.g., a computer) which discovers that a graphical button is“underneath” the touch, and therefore communicates this information tothe control processor to use a “button” haptic response for this touch.As the user presses down on the haptel, the control processor respondswith a feedback force which increases as the surface is depressed untilthe position reaches a certain threshold, at which point the feedbackforce is quickly reduced. This causes the applied force to momentarilyexceed the feedback force, which results in the quick downward movementof the haptel surface. In this way a “clicking” sensation is conveyed tothe user. Preferably, the computer is continually informed of the stateof the touch so that when the haptel reaches the bottom of its travel,the computer executes the action represented by the graphical button anddisplays the button in its activated state.

If the graphical button is disabled, the computer has the controlprocessor use a “disabled button” haptic response. In this response thefeedback force increases with position at a higher rate than the“button” response with no force drop-off. This creates the sensation ofan unyielding surface which informs the user than the action representedby the graphical button cannot be initiated.

The preceding descriptions assume that each touch falls within thebounds of a single haptel, but this need not be the case. If thetouchable area of the device is mapped to a GUI in which interfaceelements can be placed anywhere, some will happen to be located on theedge between two haptels or the vertex where four haptels meet. A touchon such a control is therefore likely land on more than one haptel. Such“border touches” can be transparently handled by the device. The firststep is to merge related touches. If two touches appear simultaneouslyon adjacent haptels a short distance apart, the device can safely inferthat the touches are really a single touch on the border between thosetwo haptels. Similar inferences can be made for touches that appearsimultaneously near the vertex of any number of haptels.

Once the set of haptels is determined, the haptels are managed in acoordinated fashion. The center of the touch is computed, preferably byweighting each touch location by the force applied to that haptel, andthen dividing by the total force applied to the haptels involved.Likewise, the collective surface position, velocity, and accelerationare computed, preferably by weighted average of the haptels involved.Other weightings are possible, including equal weighting of values. Theapplied force measurements of the haptels involved may be summed tocompute the total force applied. The haptic response is then computedfrom these collective measurements in much the same way they would becomputed for a single haptel, resulting in a collective feedback force.This feedback force is distributed across the haptels involved in thetouch in proportion to the amount of the total applied force lands oneach haptel. In addition, a restoring force pulls the haptels towardsthe collective position to prevent surfaces from drifting apart due tomeasurement errors and other factors. As a result, the total feedbackforce is effectively distributed across the haptels involved in thetouch, and the haptel's surfaces will have similar position, velocity,and acceleration. This provides the illusion that a single surface waspressed, making the coordinated nature of the touch undetectable by theuser.

Not only can such device coordinate a fixed set of haptels, but it canalso transparently add and remove haptels from the coordination set overtime. This is necessary during “dragging” operations in which touchesmove across the device. When a touch gets close to another haptel, thenewly-added haptel is added to the coordination set. This has the effectof causing its surface to become flush with the haptels already involvedin the touch. Preferably, this is done without affecting the feel of thetouch in progress. When the touch moves far enough away from a givenhaptel, that haptel is removed from the coordination set, leaving itfree to participate in another touch.

This coordination effectively makes the haptels' gridded natureinvisible to the user and to software applications. The computerspecifies the response for a touch in a declarative fashion, and thedevice ensures that this response will be generated regardless of wherethe touch falls, how many haptels are involved in the touch, or whetherthe touch moves. Device-specific information provided to the computermight include the minimum allowed distance between independent touches,so that the computer can separate controls designed for simultaneous useappropriately or give feedback to the user when one touch ventures tooclose to another.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is not intended to be in any way limiting. Otheraspects, inventive features, and advantages of the present invention, asdefined solely by the claims, will become apparent in the non-limitingdetailed description set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings. In the drawings, relatedfigures have the same number but different alphabetic suffixes.

FIG. 1 is a schematic exploded perspective representation of a portionof one embodiment of the invention, showing parts which comprise ahaptel moving assembly.

FIG. 2 is a schematic exploded perspective representation of a portionof one embodiment of the invention, showing a haptel moving assembly andconstraint pins.

FIG. 3 is a schematic exploded perspective representation of a portionof one embodiment of the invention, showing parts which comprise ahaptel stationary assembly.

FIG. 4 is a schematic exploded perspective representation of a portionof one embodiment of the invention, showing a haptel stationary assemblymounted to a support plate.

FIG. 5A is a schematic exploded perspective representation of a portionof one embodiment of the invention, showing the parts and assemblieswhich comprise a haptel.

FIG. 5B is a schematic perspective representation of a portion of oneembodiment of the invention, showing a haptel.

FIG. 6A is a schematic exploded perspective representation of oneembodiment of the invention, showing parts and assemblies which comprisea haptel grid with a flexible overlay and a hand rest.

FIG. 6B is a schematic perspective representation of a portion of oneembodiment of the invention, showing a haptel grid with a flexibleoverlay and a hand rest.

FIG. 7 is a schematic of a circuit for measuring haptel surfaceposition.

FIG. 8 is a schematic of a circuit for driving a haptel actuator.

FIG. 9 is a block diagram showing the elements of one embodiment of theinvention.

FIG. 10 is a flow chart representation of a method for controlling theapparatus.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DETAILED DESCRIPTION OF THE INVENTION

The following is intended to provide a detailed description of anexample of the invention and should not be taken to be limiting of theinvention itself. Rather, any number of variations may fall within thescope of the invention which is defined in the claims following thedescription. In addition, the following detailed description has beendivided into sections, subsections, and so on, to highlight the varioussubsystems of the invention described herein; however, those skilled inthe art will appreciate that such sections are merely for illustrativefocus, and that the invention herein disclosed typically draws itssupport from multiple sections. Consequently, it is to be understoodthat the division of the detailed description into separate sections ismerely done as an aid to understanding and is in no way intended to belimiting.

Haptel Description

FIG. 1 illustrates various aspects of one embodiment of a haptelaccording to the present invention and exemplified by a haptel 500.Haptel 500 includes, primarily, two assemblies: a moving assembly 100and stationary assembly 300. FIG. 1 illustrates an exploded perspectiveview of the parts of moving assembly 100.

An XY sensor 116 is attached to the top of a surface 102, which is inturn coupled to a coil holder 104. Edges of XY sensor 116 are preferablyfolded around surface 102 and fixed into place by glue, for example. AnXY cable 118 is provided to couple XY sensor 116 to an XY interface (notshown), to be described later. XY sensor 116 may be implemented using,for example, a four wire resistive film touch sensor. An upper bearing106 and a lower bearing 110 preferably fit closely around coil holder104 and are held in by glue, for example. Upper bearing 106 isrotationally aligned with coil holder 104 such that bearing pin holes106 a are aligned with coil pin holes 104 a, constituting constraint pinholes 100 a. Magnet wire 108 is wound around coil holder 104 betweenupper bearing 106 and lower bearing 110. Magnet wire ends 108 a arerouted through a lower wire hole 104 b, through the interior of coilholder 104, and through an upper wire hole 104 c. Magnet wire ends 108 aare electrically coupled to a coil cable 114. Magnet wire ends 108 a aremechanically but not electrically coupled (e.g., non-consecutivelyglued) to the top of coil holder 104 for purposes of strain relief.

Surface 102 and a coil holder 104 may be made, for example, of anon-ferromagnetic material with good heat conductivity (e.g., 6110-T6aluminum alloy). Preferably, the interior sides of coil holder 104 arepainted black and the interior top is painted white. Upper bearing 106and lower bearing 110 are made of a low friction material, such aspolytetrafluoroethylene (PTFE). Coil cable 114 and XY cable 118 may be,for example, high-flexibility multi-conductor shielded cables, withjacket and shield removed from the flexing portion. Magnet wire 108 maybe, for example, standard insulated magnet wire.

FIG. 2 illustrates an exploded perspective view of moving assembly 100and constraint pins 200. This figure shows how constraint pins 200 fitinto constraint pin holes 100 a. Constraint pins 200 may be, forexample, metal cylinders with a smooth surface, such as spring steelwire.

FIG. 3 illustrates an exploded perspective view of stationary assembly300. In the embodiment shown in FIG. 3, a flux disk 306 is attached to amagnet 304, which in turn is attached to a base 302. A proximity sensor308 is electrically coupled to a position cable 310, which passesthrough flux disk hole 306 a, magnet hole 304 a and base cable hole 302a. Position cable 310 couples proximity sensor 308 to a position circuit(not shown), to be described later. Position cable 310 is preferably ashielded four conductor cable. Preferably, the bottom of proximitysensor 308 is flush with the top of flux disk 306 and secured (e.g.,glued into place). A spring 312 is affixed to flux disk 306 surroundingproximity sensor 308. A lower base bearing 314 preferably fits closelyinside a midsection 316, and is secured (e.g., glued into place).Midsection 316 fits around the top of base 302 and is rotationallyaligned to be square with base 302. Upper base bearing 318 fits closelyinside top section 320, and is rotationally aligned such that bearingslots 318 a are aligned with top slots 320 a, constituting constraintpin slots 300 a. Top section 320 fits closely around the top ofmidsection 316 and is rotationally aligned to be square with midsection316 and base 302.

Magnet 304 is preferably a high-strength permanent magnet. Base 302,flux disk 306 and midsection 316 are preferably made of a ferromagneticmaterial with good permeability (e.g., steel alloy 12L14). The top andsides of flux disk 306 are preferably painted black before assembly. Topsection 320 is preferably made of a non-ferromagnetic material (e.g.,6110-T6 aluminum alloy). Upper base bearing 318 and lower base bearing314 are preferably made of a low-friction material (e.g., PTFE).Proximity sensor 308 may be implemented using, for example, a reflectiveproximity sensor containing an LED (not shown) and a phototransistor(also not shown).

FIG. 4 illustrates an exploded perspective view of a haptel stationaryassembly 300 mounted to a support plate 402. Support plate 402 is madeof a rigid material with good heat conductivity, such as aluminum plate.It will be noted that FIG. 4 shows only a portion of support plate 402.Mounting hardware 404 may consist of two sets of machine screws, nutsand washers, for example. In one embodiment, the machine screws arerouted through two diagonally opposite holes in haptel stationaryassembly 300 and through support plate 402, and are fastened securely tothe other side using nuts and washers. Position cable 310 is routedthrough a center hole in support plate 402. The hole pattern in supportplate 402 should match the hole pattern in base 302.

FIG. 5A illustrates an exploded perspective view of the parts andassemblies of a haptel such as haptel 500. FIG. 5B illustrates aperspective view of haptel 500. Moving assembly 100 fits insidestationary assembly 300. Moving assembly 100 is preferably aligned suchthat constraint pin holes 100 a are aligned within constraint pin slots300 a, making surface 102 of moving assembly 100 square with stationaryassembly 300. Constraint pins 200 are glued into constraint pin holes100 a, fitting within, but not affixed to, constraint pin slots 200 a.Coil cable 114 and XY cable 118 are routed through the remaining cornerholes in stationary assembly 300 and support plate 402.

Grid Description

FIG. 6A illustrates an exploded perspective view of the parts includedin grid assembly 600. FIG. 6B illustrates a perspective view of gridassembly 600.

Unified support plate 602 is shown in FIG. 6A as having nine haptels(e.g., haptel 500) in a 3×3 grid configuration. Unified support plate602 replaces support plate 402 for all haptels in the grid. Therectangularly arranged set of nine haptels is referred to as a haptelgrid 604. The bolts, coil cables, position cables, and XY cables of allhaptels go through appropriately positioned holes in unified supportplate 602. In addition, there is a hole in the unified support platebeneath each base air hole 302 b. Preferably, grid overlay 606 isaffixed to haptel grid 604 at the centers of the haptel surfaces. Gridoverlay 606 is a thin, slick, flexible and stretchable material, such as0.010″ thick urethane elastomer sheet with 60 Shore A durometerhardness. Hand rest 608 is affixed to support plate 602 with grid feet610. Hand rest 608 is preferably made of injection molded plastic, whilegrid feet 610 are preferably plastic with standard screws centrallyembedded therein. The screws projecting from grid feet 610 preferablythread into holes in vertical supports 608 a. The height of verticalsupports 608 a ensures that the upper surface of hand rest 608 is flushwith the upper surface of haptel grid 604 when assembled.

System Description

FIG. 9 illustrates a block diagram depicting the functional elements ofan embodiment of the present invention. An input/output (I/O) device 900includes haptels 500(1)-(N), position circuits 700(1)-(N), actuatorcircuits 800(1)-(N), XY interfaces 912(1)-(N) and a control system 902.Control system 902 includes an analog input card 906, two serial cards910, a digital output card 908, and a control processor 904. Functionalelements of haptels 500(1)-(N) are shown as a group containing magnetwires 108(1)-(N), proximity sensors 308(1)-(N) and XY sensors116(1)-(N).

It will be noted that the variable identifier “N” is used in severalinstances in FIG. 9 to more simply designate the final element (e.g.,haptel 500 (N), XY sensor 116(N), and so on) of a series of related orsimilar elements (e.g., haptels 500(1)-(N), XY sensor 116 (1)-(N), andso on). The repeated use of such variable identifiers is not meant toimply a correlation between the sizes of such series of elements,although such series may be equal in extent. The use of such variableidentifiers does not require that each series of elements have the samenumber of elements as another series delimited by the same variableidentifier. Rather, in each instance of use, the variable identified by“N” may hold the same or a different value than other instances of thesame variable identifier. For example, haptel 500(N) may be the ninth ina series of haptels, whereas XY sensor 116(N) may be the forty-eighth XYsensor in a series of XY sensors. In the preferred embodiment, N equalsnine for all series.

Each one of haptel proximity sensors 308(1)-(N) is coupled via acorresponding one of position cables 310(1)-(N) to a corresponding oneof position circuits 700(1)-(N) as described in FIG. 7. The output ofeach one of position circuits 700(1)-(N) is coupled to an input ofanalog input card 906, which is installed in control system 902 andcommunicates with control processor 904 via a communications channelsuch as PCI bus 922. Analog input card 906 is preferably a high-speeddata acquisition card with a number of inputs corresponding to thenumber of haptels in haptel grid 604 and may employ devices such asthose available from National Instruments of Austin, Tex. under thetrade designation PCI-6023E. Magnet wires 108(1)-(N) couple theirrespective haptel via one of coil cables 114 to the outputs of arespective one of actuator circuits 800(1)-(N), described in FIG. 8. Theinputs of each of actuator circuits 800(1)-(N) are coupled to theoutputs of digital output card 908 which is shared by actuator circuits800(1)-(N). Digital output card 908 is installed in control system 902and communicates with control processor 904 via a communications channelsuch as a device PCI bus 922. Digital output card 908 preferablyprovides at least 14 bits of parallel output, and may employ devicessuch as are available under the trade designation DIO-32HS, fromNational Instruments.

Each one of haptel XY sensors 116(1)-(N) is coupled via a correspondingone of XY cables 118(1)-(N) to a corresponding one of XY interface912(1)-(N). Each one of XY interface 912(1)-(N) digitizes readings froma corresponding one of XY sensors 116(1)-(N) and provides an interface(e.g., RS-232 serial interface) to this data. XY interfaces 912(1)-(N)may be implemented using devices such as those under the tradedesignation CS6000 and available from CyberTouch of Newbury Park, Calif.The serial port of each of XY interfaces 912(1)-(N) is coupled to acorresponding serial port on serial cards 910. Each serial card haseight ports, thus two serial cards are required to support the nine XYinterfaces in this embodiment. Serial cards 910 are installed in controlsystem 902 and communicates with control processor 904 via acommunication channel such as PCI bus 922 using, for example, devicessuch as the National Instruments device designated PCI-232/8.

Control processor 904 is connected via serial link 914 to a computer916. Typically, control processor 904 and computer 916 are bothappropriately programmed general purpose processors. In one embodiment,computer 916 is a computer system such as a personal computer system.Other embodiments may include different types of computer systems.Computer systems may be found in many forms including but not limited tomainframes, minicomputers, workstations, servers, personal computers,notepads and embedded systems. A typical computer system includes atleast one processing unit, associated memory and a number ofinput/output (I/O) devices. A computer system processes informationaccording to a program and produces resultant output information via theI/O devices. A program is a list of internally stored instructions suchas a particular application program and/or an operating system. Asoftware module may include a program. The programs that control theoperation of a computer system are commonly referred to as softwareapplications or simply software. Preferably, control processor 904 andcomputer 916 are implemented using a processor such as an Intel PentiumIII operating at 550 MHz.

Operation

The operation of a device such as I/O device 900 is now described. Themechanical operation of a haptel is first described, followed by adescription of the operation of a proximity sensor and actuator. Theoperation of control system 902 is then described.

Haptel Mechanical Operation

In operation, the top of XY sensor 116 is pressed with a pointingelement such as a finger or stylus causing moving assembly 100 to moveup and down. Constraint pins 200 limit the vertical travel of movingassembly 100 and keep moving assembly 100 from rotating relative tostationary assembly 300. Spring 312 applies an upward force to movingassembly 100 which returns moving assembly 100 to an upper limit oftravel when not depressed.

When XY sensor 116 is pressed anywhere other than at the exact center, atorque is applied to moving assembly 100. This torque causes movingassembly 100 to tilt and applies a normal force to the bearings thatincreases friction. To minimize this tilt, the gap between the inner andouter bearings is kept small, preferably less than a fifth of amillimeter, for example. The vertical spacing between the upper andlower set of bearings further reduces the tilt angle. Friction isminimized by making the bearings from a material having a very lowcoefficient of friction (e.g., PTFE). Even in the case of a touch at thefar corner of surface 102, friction is preferably kept below 10% of theapplied force. Minimizing off-axis friction ensures that the dynamics ofI/O device 900 are kept as independent of touch location as possible.

Haptel 500 is designed such that moving assembly 100 can move freelywith little tilt or rotation. This allows adjacent haptels to bepositioned with minimal gaps between the edges of their surfaces and yetavoid contacting one another during use. Small gaps also tend to makethe gaps between haptels less noticeable to the user. Preferably, baseair hole 302 b is present so air can move more freely in and out of theinterior of haptel 500 during use. If not included, motion can beimpeded as a result of the air escape between the inner and outerbearings.

When adjacent haptels are touched simultaneously, the haptels tiltslightly towards one another, but are prevented from touching due inpart to the design and manufacturing tolerances selected. The seamsbetween the haptel's surfaces are preferably such that such seams arelargely invisible to the user. Grid overlay 606 also helps to make theseams between the haptels less noticeable. The material of grid overlay606 is preferably somewhat stretchable. This allows adjacent haptelsurfaces (e.g., surface 102) to be at different heights without thematerial of grid overlay 606 overly restricting their motion. Thestretchiness required depends in part on the travel of the haptels andthe size of their surfaces (e.g., surface 102).

A vertical travel of a few millimeters is adequate to simulate thehaptic response of a key press, although the travel of a haptel'ssurface can vary from on the order of about 0.1 mm to about 2 cm (ormore). The size of surface 102 is preferably as small as is feasible.This, in part, allows for more simultaneous touches and a smallerminimum distance between touches. In one embodiment, the size of surface102 (for each haptel) preferably corresponds to that of a pixel. Themass of moving assembly 100 is preferably minimized in order to maximizethe acceleration for a given actuator force and enable a greater rangeof useful haptic effects.

Precise manufacture of the haptel is important because the fingertip isvery sensitive to shape and texture. The haptel surfaces are preferablywell aligned with each other at both extents of their travel. Verticalalignment within 0.1 mm is preferred.

In general, the haptel surfaces can be of any size, and need not besquare or even all of the same shape, so long as they are tiled withminimal gaps between their edges. This includes, for example,rectangular, triangular or hexagonal haptels, but other irregular ornon-periodic tilings are possible. The overall shape of the touchablearea can be any shape, such a rounded rectangle, an ellipse, or anirregular shape. Depending on the shape of the haptel surfaces and theshape of the touchable area, some portion of the haptels on the edges ofthe tiling might be unused, and perhaps covered with the hand rest toprevent use or indicate the touchable portion.

Preferably, high flexibility wire is used for coil cable 114 and XYcable 118 because of the relatively large motion of the moving assemblyrelative to the short length of the wires. The wire is preferably veryfinely stranded wire with silicone insulation. The wires should notsignificantly impede the motion of the moving assembly. Suitable partsare exemplified by devices with trade designation AS999-30-2SJ for thetwo conductor coil cable and AS-155-28-5SJ for the five conductor XYcable, available from Cooner Wire of Chatsworth, Calif.

Proximity Sensor Operation

Proximity sensor 308 is preferably designed to provide high resolutionand bandwidth at a reasonable cost. High resolution increases thefidelity of the haptic display in general and in particular improves thequality of velocity and acceleration measurements derived from theoutput of proximity sensor 308. Proximity sensor 308 preferably providesa resolution of 0.05 mm, and most preferably a resolution of about 0.01mm.

The position of moving assembly 100 within its range of travel ismeasured by proximity sensor 308. In one embodiment, proximity sensor308 contains an infrared LED and an infrared phototransistor in the sameenclosure. In such a device, the LED and phototransistor point upwardand are optically isolated from each other within proximity sensor 308.An example of such a device is available from Optek of Carrollton, Tex.under the trade designation OPB-710.

Position circuit 700, shown in FIG. 7, interfaces proximity sensor 308to analog input card 906. Resistor RI limits the current through the LEDto an allowed value, causing the LED to emit a constant amount of light.A typical value would be 76.8 Ohm. The light emitted by the LED isreflected by the interior top of coil holder 104. Some of the reflectedlight is received by the phototransistor. The current through thephototransistor is proportional to the quantity of light fallingthereon. Resistor R2 converts this phototransistor current to a voltagewhich forms the input of low pass filter 702. A typical value for R2 is2.21 kOhm. Low pass filter 702 is a voltage-controlled voltage-source2-pole Butterworth filter with a 3 dB roll-off point at 1.3 kHz and gainof 1.59. Typical component values are 12.1 kOhm for resistor R3, 0.01microFarad for capacitors C1, 33.2 kOhm for resistor R4, 56.2 kOhm forresistor R5, and 0.1 microFarad for bypass capacitors C2. Op amp 704 isa CMOS operational amplifier with rail-to-rail operation. Suitable partsare available from National Semiconductor of Santa Clara, Calif. undertrade designation LMC6482. The frequency roll-off of the low-pass filteris preferably lower than half of the per-channel sampling rate of theanalog input card. Additionally, it will be noted that, preferably, thephototransistor provides bandwidth commensurate with its sampling rate.Using a 12-bit resolution analog input card as analog input card 906,control system 902 can discern between moving assembly positionsseparated by about 0.01 mm.

The interior top of coil holder 104 is preferably painted to diffuselyreflect infrared light. Diffuse reflectivity ensures that thephototransistor current varies smoothly and monotonically with distance,and provides a consistent reading independent of any tilt of the movingassembly. The interior sides of coil holder 104, and the exterior topand sides of flux disk 306, are preferably painted to absorb infraredlight so that light does not reach the phototransistor through secondaryreflections. Such an embodiment provides better contrast betweenreadings at the limits of travel.

The output of position circuit 700 is usually not linear with respect tothe position of moving assembly 100. The output may be characterized asbeing approximately the inverse square of the distance between proximitysensor 308 and the inside surface of coil holder 104. This effect can becorrected, for example, by calibrating proximity sensor 308 prior touse. The moving assembly 100 is moved across its range of possiblepositions very precisely, for example, with a micrometer in steps of0.001 inch. The output of the position circuit is then measured andrecorded. Later, when a making a position measurement, the outputcorresponds to one of position circuits 700(1)-(N) and is compared tothe stored values, and the distance is computed with an interpolation ofthe two closest readings. This calibration procedure also corrects forany non-linearity in the response curve of the phototransistor.Alternatively, an equation could be fit to the calibration data to allowa direct computation of the position from a reading.

XY Sensor Operation

XY sensor 116 may be of many conventional designs, such as a resistivefilm position sensor. The effective resolution of a resistive filmposition sensor is typically greater than 100 dots per inch (dpi). Inthe scenario in which haptel grid 604 is mapped to a display (notshown), the cumulative resolution of XY sensors 116(1)-(N) on haptelgrid 604 is preferably equal to or greater than the resolution of thedisplay area mapped to I/O device 900 (e.g., as a result of the directmapping between haptel grid 604 and the display). However, there may bepixels on the display which cannot be touched by the user. For example,the resolution of computer displays is typically 75 to 100 dpi, thushaptel grid 604 can be mapped to an equal or somewhat larger displayarea when resistive film technology is used in XY sensors 116(1)-(N).

The actuation force of this type of sensor can be controlled during itsmanufacture. Unlike other uses for these touch sensors, the sensor isnot used to initiate actions but only to sense position. Because eachhaptel potentially maps to multiple user interface elements, the devicecannot determine which haptic effect to generate until the XYcoordinates of a touch are known. Preferably, the lowest possibleactuation force is thus used. For example, a value of 5 grams could beemployed.

Actuation force also comes into play when touches overlap multiplehaptels. Because applied force is spread across the haptels involved inthe touch, it is possible that the force on one or more of the haptelsis below the threshold of its XY sensor. In the worst case, a force of20 grams would have to be applied at the vertex of 4 haptels beforegetting an XY reading (using a 5 gram actuation force). While users canadapt to differences in actuation force for different parts of thedisplay, the user experience is enhanced by a more uniform and loweractuation force. An alternative is the use of sensors that exhibit areduced actuation force, such as capacitive proximity sensors.

Actuator Operation

Integral to the design of the haptel in the preferred embodiment isactuator 504. Actuator 504 is comprised of the following subset ofhaptel 500 parts: magnet 304, flux disk 306, midsection 316, base 302,magnet wire 108 and coil holder 104. Magnet 304 creates and sustains amagnetic field, which is magnetically conducted upward through flux disk306, radially outwards across the air gap between the flux disk andmidsection 316, downward through the midsection, downward into base 302,and radially inwards and upwards through the base, returning to theother side of the magnet. Thus a magnetic field of high flux density iscreated in the air gap between the flux disk and the midsection. Thisair gap is occupied by magnet wire 108 and coil holder 104, whichphysically supports the magnet wire.

The actuator is preferably a high efficiency, high-bandwidth device.High efficiency allows relatively large forces to be generated withoutoverheating. Preferably, such a device generates a force of about 2 Nfor a power consumption of under 5 W. Peak force in the preferredembodiment was selected to be large enough to adequately simulate akeypress effect. High bandwidth allows the force feedback to changequickly, improving the quality of haptic effects.

To apply a force to moving assembly 100, current is driven throughmagnet wire 108. The direction of the current flow determines thedirection of the force. To improve efficiency, a rare earth magnet withenergy density of at least about 27 million Gauss-Oersteds is preferablyemployed. Ferromagnetic material with a good permeability is preferablyused for the flux return path to further maximize the air gap fluxdensity. There are design tradeoffs between the air gap length, thesurface area and thickness of the magnet, among other factors, whichwould be apparent to a practitioner skilled in the art ofelectromagnetic actuator design.

Each of one magnet wire 108(1)-(N) is driven by an actuator circuit 800,shown in FIG. 8. In one embodiment, a digital input bus 14 bits wide isshared by all of actuator circuits 800(1)-(N). The least significant 8bits of the digital bus encodes the pulse width, the next 4 bits encodethe haptel identifier, the next bit encodes the latch control, and themost significant bit encodes the direction of the actuation (e.g.,pulling downward or pushing upward).

Comparator 805 is a 4-bit comparator. Suitable parts are available fromMotorola of Schaumberg, Ill. under trade designation MC74F85. Comparator805 continually compares the haptel identifier on the digital bus to thebinary number encoded on identifier switch array 815. Identifier switcharray 815 is a 4 position DIP switch. Suitable parts are available fromGrayhill of La Grange, Ill. under trade designation 76SB04. Eachactuator circuit has a unique setting on its identifier switches. Thefour bits of identification are adequate to distinguish between the ninehaptels in this embodiment. Resistor network 820 keeps comparator inputsnear ground when a switch is open. Suitable parts are available fromMatsushita Electric of Secaucus, N.J. under trade designationEXB-F6E222G.

The latch control bit couples to the =IN input of comparator 805, andinverter 810 couples the logical negation of the latch control bit tothe <IN input. When the latch control bit is high, the P=Q output ofcomparator 805 is high when the haptel identifier on the digital busmatches the number encoded on the identifier switches. When the latchcontrol bit is low the P=Q output is low regardless of the haptelidentifier on the digital bus. Inverter 810 is a standard NOT gate.Suitable parts are available from Motorola under trade designationMC74F04.

When the P=Q output on comparator 805 is high, the pulse width data ispassed through latch 825 and the direction bit is passed through latch830. The data should remain on the digital bus until the latch bit goeslow. In this way, the pulse width and direction data remain latched, andthe actuator circuit can drive the magnet wire until a new value isassigned. Latch 825 and latch 830 are 8-bit latches. Suitable parts areavailable from Texas Instruments of Dallas, Tex. under trade designationSN74F573.

Clock 835 is a CMOS oscillator which generates a clock signal at a highfrequency, preferably at least 4 MHz. Suitable parts are available fromEpson America of Torrance, Calif. under trade designation SG531 P.Inverters 840 and 850 are standard NOT gates, such as the MotorolaMC74F04. Inverters 840 invert the clock signal and delay the propagationof the signal. There is a brief time period during at the very beginningof each clock cycle when both inputs to AND gate 845 are high. AND gate845 is a standard AND gate, such as the Motorola MC74F08. The output ofinverter 850 is a delayed version of the original clock, due to the evennumber of inverters. Thus the pulse from AND gate 845 comes at the veryend of the clock cycle which drives counters 855 and 860.

Counters 855 and 860 are 4-bit synchronous counters cascaded to form an8-bit counter. Suitable parts are available from Texas Instruments undertrade designation SN74F163A. This counter freely cycles from 0 to 255,driven by the delayed clock signal. Comparators 865 and 870 are 4-bitcomparators cascaded to form an 8-bit comparator. Suitable parts areavailable from Motorola under trade designation MC74F85. The P>Q outputof comparator 870 is high when the latched pulse width value is strictlygreater than the current synchronous counter value. Thus the width ofthe P>Q pulse is proportional to the pulse width value. If the pulsewidth value is 0, the P>Q output is never high. If the pulse width valueis 255, the P>Q output is high for 254 out of every 255 clock cycles.

The output of AND gate 845 latches the outputs of comparator 870 andlatch 830 into latch 875 at the end of each clock cycle. In this way,the output of comparator 870 is guaranteed to be valid, taking intoaccount the propagation delays in the synchronous counters and thecomparators from the beginning of the clock cycle. Latch 875 is an 8-bitlatch. Suitable parts are available from Texas Instruments under tradedesignation SN74F573.

Motor driver chip 885 uses the pulsing comparator output to enable anddisable its outputs. When 1,2EN is high, the outputs are switched on andcurrent can flow. When 1,2EN are low, the outputs are in ahigh-impedance state and no current flows through magnet wire 108. Thedirection of the current flow is determined by the 1A and 2A inputs.When 1A is high, the IY output is at 12 volts, and when 1A is logic low,the 1Y input is at ground. Likewise for the 2A input and 2Y output. Dueto inverter 880, the 1A and 2A inputs are always logically opposite.Thus when the direction bit is high, the 1Y output is at 12 Volts andthe 2Y output is at ground, causing current to flow through the magnetwire in one direction. When the direction bit is low, the 1Y output isat ground and the 2Y output is at 12 Volts, causing current to flow inthe other direction. Flyback diodes 890 prevent the inductance of themagnet wire from damaging the motor driver chip when the outputs areswitched off. Coil cable 114 couples actuator circuit 800 to magnet wire108. Motor driver chip is high-current driver chip. Suitable parts areavailable from Texas Instruments under trade designation L293D. Flybackdiodes 890 are high-current fast-recovery diodes, such as the UF1002Tmade by Vishay Intertechnology of Malvern, Penn.

Actuator circuit 800 is just one way to interface a processor to anactuator. Many different circuits are possible, depending on thebandwidth and accuracy desired and the type of actuator beinginterfaced, among other factors. For example, a digital-to-analogconverter could be used in combination with a linear amplifier to drivean electromagnetic actuator.

As a result of coil inductance, the relationship between the forcegenerated by a given haptel and the width of the pulse generated byactuator circuit 800 may not be linear. Actuator circuits 800(1)-(N) aretherefore calibrated prior to use by measuring the force actuallygenerated by each output value, and storing that measurement. Later,when a certain output force is desired, the correct output value to useis calculated by interpolating between the closest desired outputvalues.

Substantial energy is dissipated by magnet wires 108(1)-(N) at its peakforce output. To minimize heat buildup in the haptel, coil holder 104and surface 102 are preferably made of a material with high heatconductance. This allows heat to be conducted upward to surface 102,where the heat can radiate away. Heat also radiates to stationaryassembly 300, which can be configured to act as a heat sink incombination with unified support plate 602.

Control System Operation

FIG. 10 illustrates a flow diagram of the operation of I/O device 900according to one embodiment of the present invention. The flow diagramof FIG. 10 describes the actions performed in running control system 902(e.g., by software). The management of I/O device 900 is preferablyhandled as a real-time task, so a separate control processor ispreferably used instead of running control software on computer 916. Thecontrol loop's frequency can be an important factor in the operation ofI/O device 900. The computational performance of control processor 904,the speed of input and output operations, and the degree of optimizationof the control software all affect the frequency (and thusresponsiveness) attainable on a specific system. In one embodiment, acontrol loop frequency of 10 kHz is used.

The process begins with the reading of the haptels' position sensors(step 1002). Analog input card 906 is preferably programmed duringinitialization to sample all inputs cyclically (e.g., for a system withnine haptels, and so, nine inputs, this would be at an aggregatefrequency of 90 kHz providing a per haptel sampling rate of 10 kHz, andthus matching the control loop frequency). The sampling rate ispreferably selected to exceed the bandwidth of proximity sensors308(1)-(N) and the maximum pass frequency of position circuits700(1)-(N) by a factor of two. The stored calibration values are used tocompute the calibrated position value of each moving assembly (e.g., inmillimeters). These calibrated position values may then be stored forlater use.

Derived measurements are computed for each haptel using position values(step 1004). The current velocity is computed by differentiating theposition values. The current acceleration is computed by differentiatingvelocity values. In both cases, the result is low-pass filtered in orderto reduce noise. The net force acting on moving assembly 100 is computedby dividing the acceleration of moving assembly 100 by the mass ofmoving assembly 100. The force applied to XY sensor 116, referred toherein as the applied force, is the result of subtracting the actuatorforce, the spring force and the weight of the moving assembly from thenet force. The actuator force is known because the control computer setsthe actuator force, the spring force is computed from the springconstant and the current position of moving assembly 100, and the weightof the moving assembly was measured prior to assembly. Thus, thevelocity, acceleration and applied force for each haptel can be computedfrom position measurements. Although derived measurements may have morenoise, higher latency and less bandwidth than the original positionmeasurements, such measurements are still adequate to allowimplementation of the methods described herein.

Next, control processor 904 reads XY data from serial ports (step 1006).XY interfaces 912(1)-(N) send updated XY position values relativelyinfrequently when compared to the relatively high frequency at which thecontrol loop runs, and so there is normally no data waiting. If data iswaiting, control processor 904 reads the updated value and stores thatvalue in memory. The value either encodes the location of a touch, orencodes that there is no touch applied to the sensor. The XY values areconverted to a universal coordinate system using the position of thehaptel within the haptel grid to offset the haptel's XY sensor reading.

Control processor 904 examines the incoming XY data for new touches(step 1008). If a touch has appeared on a haptel since the last datapoint, three cases are possible. If this haptel is already part of acoordinated touch, no action is taken. If the haptel is not currentlypart of a coordinated touch, control processor 904 determines if a touchrecently appeared within a predetermined distance of the haptel andwithin a predetermined time period. If such a touch is found, then thistouch is presumed to be part of the same touch, and this haptel is addedto the coordinated touch. If no such touch is found, then a newcoordinated touch is started with this haptel as the only member.

Control processor 904 updates the collective measures for allcoordinated touches (step 1010). For each coordinated touch, thecollective position is computed as an average of the vertical positionvalues for each haptel belonging to the touch, preferably with eachvalue weighted by the applied force for that haptel, as derived earlier(step 1004). Thus haptels with no applied force do not contribute to thecollective position measurement. Other weightings are possible,including equal weighting. Likewise, the collective velocity andcollective acceleration are computed as a weighted average of thesevalues for the haptels that constitute the coordinated touch. Thecollective XY location of the touch is also preferably a weightedaverage of the constituent XY readings. The collective applied force isa total of the applied force for all the haptels involved in thecoordinated touch.

The collective XY measurements for a coordinated touch are evaluated(step 1012) to determine if haptels need to be added to, or removed fromthe coordinated touch. If the collective XY of a touch is less than afirst predetermined distance of from a nearby haptel, that haptel isadded to the coordinated touch, if the haptel is not already a member ofanother coordinated touch. If the collective XY of a touch is greaterthan a second predetermined distance from a haptel already a part of thecoordinated touch, that haptel is removed from the coordinated touch.The second predetermined distance is preferably greater than the firstpredetermined distance, and the difference between these two values ispreferably large enough to prevent a given haptel from being quicklyadded to and removed from a touch if the given haptel's distance fromthe touch is close to these values. The first and second predetermineddistances should be large enough that the haptels being added andremoved are not currently being touched. In one embodiment, a valuegreater than half the size of the intended touching object, such as thefingertip, is often adequate.

The current state of coordinated touches is periodically sent tocomputer 916 over serial link 914 (step 1014). Computer 916 ispreferably updated with a frequency on the order of hundreds of timesper second. Computer 916 is preferably sent only the collectivemeasurements of XY, position, velocity, acceleration and applied force.Serial link 914 is preferably checked for incoming data from computer916 containing instructions on the haptic effects to use for eachcoordinated touch (step 1016). Typically, haptic effect commands aresent after a new coordinated touch starts, and only periodically oncethe touch is in progress. The haptic effect might be changedsubsequently depending on the state of the software executed by computer916. For example, after a user has started to press a virtual button,computer 916 can disable the button. The new haptic effect is preferablyimplemented as soon as it is received. Haptic effect command can bedesignated to simply assign one of many built-in haptic effects to thecoordinated touch or to define a custom haptic effect (e.g., by mixingtogether built-in effects, transmitting a force response curve, ordownloading executable machine codes which implement an effect).

The collective feedback force for each coordinated touch is thencomputed based on the collective measurements derived earlier using thehaptic effect command sent from the computer (step 1018). For a simple“button” response, for example, the feedback force can be primarilycomputed based on the collective position. At 0.0 mm travel the force isabout 0.05 Newton, which increases linearly to 0.75 Newton over the next0.3 mm of travel, then decreases to 0.10 Newton over the next 0.5 mmtravel, and finally increases to 2.0 Newtons over the next 1.0 mm oftravel, and stays at 2.0 Newtons over the remainder of travel. For asimple “disabled button” response, for example, the feedback force is0.05 Newton at 0.0 mm of travel, increasing linearly to 2.0 Newtons overthe next 0.6 mm of travel, and staying at 2.0 Newtons over the remainderof the travel. These are just two examples of haptic effects, amultitude of which are possible.

The feedback force for each haptel is computed by distributing thecollective feedback force across all of the haptels in the coordinatedtouch in proportion to the amount of the applied force on each haptel(step 1020). If the applied force of a given haptel is, for example,half of the total applied force for the collective touch, then thefeedback force for that haptel will be half of the collective feedbackforce. Different haptels in the same coordinated touch can havedifferent feedback force values, because they each have differentapplied forces, but the total of these forces will equal the feedbackforce for the collective touch.

The restoring force for each haptel is computed based on each haptel'sdistance from the collective position (step 1022). Due to errors inmeasurements and other effects, the surfaces of the haptels in acoordinated touch can drift apart vertically. Thus a restoring force isapplied which pulls each haptel in a coordinated touch back towards thecollective position, proportional to its distance from that position.The restoring force is preferably greater than 2.0 Newtons permillimeter. The upper limit for this value depends on the control loopfrequency and the resolution of the position measurement, among otherfactors. Since the haptel positions are distributed around thecollective position, this restoring force does not add noticeable netforce to the coordinated haptic effect. Additionally, damping can beadded in proportion to the relative velocity of a haptel compared to thecollective velocity. This prevents haptels from oscillating around theaverage position.

The net force for each haptel is computed, then converted to the correctactuator circuit output value and set via the digital output card (step1024). The net force for each haptel is simply the sum of that haptel'sdesired feedback force and that haptel's restoring force. The actuatorforce is computed by subtracting the effect of the spring at the currentposition and the weight of the moving assembly from the net force. Theactuator force is converted to the correct output value using thecalibration table recorded previously. The actuator circuits can beprogrammed by writing the output value to digital output card 908 by,for example, first writing out the data values, and then raising andlowering the latch bit to latch in the new data. After the actuators areset, the flow of control returns to step 1002. The program loopsindefinitely while haptic responses are being generated.

The operations referred to in FIG. 10 and elsewhere herein may bemodules or portions of modules (e.g., software, firmware or hardwaremodules). For example, although the described embodiment includessoftware modules and/or manually entered user commands, the variousexemplary modules may be application specific hardware modules. Thesoftware modules discussed herein may include script, batch or otherexecutable files, or combinations and/or portions of such files. Thesoftware modules may include a computer program or subroutines thereofencoded on computer-readable media.

Additionally, those skilled in the art will recognize that theboundaries between modules are merely illustrative and alternativeembodiments may merge modules or impose an alternative decomposition offunctionality of modules. For example, the modules discussed herein maybe decomposed into submodules to be executed as multiple computerprocesses. Moreover, alternative embodiments may combine multipleinstances of a particular module or submodule. Furthermore, thoseskilled in the art will recognize that the operations described inexemplary embodiment are for illustration only. Operations may becombined or the functionality of the operations may be distributed inadditional operations in accordance with the invention.

Thus, an I/O device such as that described herein provides a natural wayof interacting with a computer by employing direct mapping that providesfor multiple simultaneous inputs and dynamic haptic feedback, in orderto enhance a user's interactive experience.

While particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein, changes and modifications may be madewithout departing from this invention and its broader aspects and,therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this invention.

Moreover, while the invention has been particularly shown and describedwith reference to these specific embodiments, it will be understood bythose skilled in the art that the foregoing and other changes in theform and details may be made therein without departing from the spiritor scope of the invention. For example, there can be an arbitrary numberof haptels in the grid. The touchable surface can be divided intomultiple distinct areas whose tilt and orientation is varied relative toeach other or fixed by the design. Moreover, the individual haptels neednot be flat, so long as the surfaces of neighboring haptels form acontinuous surface when tiled together. For example, the haptel surfacescan each have a three-dimensional quality (e.g., a slight sphericalcurve). Such haptels could thus be tiled such that the touchable areaforms a section of a sphere, ellipsoid, or other three-dimensionalsurface. The proximity sensor can be replaced by a linear opticalencoder or other type of position sensor. Position measurement can bederived from other sensors, such as by integrating the signal from avelocity sensor. Multiple sensors of different types can be combined,such as a position sensor and an accelerometer, for example. A forcesensor can be added to the design, such as a piezoelectric sensor, orother type of force sensor. The spring can be removed and the actuatorpowered to provide an upward force adequate to keep the haptel at itsupper travel limit (or some other position) when no force is applied.The actuator can be replaced with another type of actuator, such as amoving magnet actuator. Alternative bearing designs may also be used(e.g., other types or combinations of bearing materials, or rollingbearing designs, among other possibilities). The haptels can becoordinated in other ways, such as making the feedback force for onehaptel or set of haptels equal to the applied force of a differenthaptel or set of haptels, and vice versa. The touch location sensor canbe replaced by another type of sensor, such as a capacitive proximitysensor. The grid of haptels can be covered by a single, flexible touchlocation sensor, or by multiple, possibly overlapping, flexible touchposition sensors. Each haptel could contain multiple touch locationsensors, so that more than one touch could be distinguished within thebounds of each haptel. The functionality of the control processor can beimplemented by the computer.

Accordingly, the scope of the invention should be determined not by anyof the embodiments illustrated, but with reference to the appendedclaims and their legal equivalents.

What is claimed is:
 1. An input/output device comprising: a plurality ofhaptic elements, wherein each one of said haptic elements comprises acontact surface, said contact surfaces define a surface, and each hapticelement of said haptic elements is configured to provide a haptic effectat a contact surface of said each haptic element upon said contactsurface of said each haptic element being touched; a plurality ofsensors, wherein each one of said sensors is coupled to a contactsurface of a corresponding one of said haptic elements, and each one ofsaid sensors is configured to generate information upon said contactsurface of said corresponding one of said haptic elements being touched;and a processor, wherein said processor is coupled to said sensors andsaid haptic elements, and said processor is configured to control saidhaptic effect produced by at least one of said haptic elements based oninformation received from a sensor of said sensors coupled to said atleast one of said haptic elements.
 2. The input/output device of claim1, wherein said sensor is further configured to generate informationbased upon a proximity of a pointing device to said contact surface. 3.The input/output device of claim 1, wherein said sensor is a resistivefilm sensor.
 4. The input/output device of claim 1, wherein said sensoris a capacitive proximity sensor.
 5. The input/output device of claim 1,wherein said processor is configured to control a haptic effect producedby certain ones of said haptic elements based on information receivedfrom ones of said sensors corresponding to said certain ones of saidhaptic elements, and said certain ones of said haptic elements are onesof said haptic elements being touched.
 6. The input/output device ofclaim 5, wherein said processor is further configured to causecontiguous haptic elements of said certain ones of said plurality ofhaptic elements to collectively produce the haptic effect.
 7. Theinput/output device of claim 5, wherein said processor is configured tocause contiguous haptic elements of said certain ones of said pluralityof haptic elements to collectively produce the haptic effect by virtueof being configured to calculate a weighted average of a forceexperienced by each one of said contiguous haptic elements.
 8. Theinput/output device of claim 5, wherein new haptic elements of saidplurality of haptic elements are added to a set of haptic elements inresponse to said new haptic elements also being touched and old hapticelements in said set of haptic elements are removed from said set ofhaptic elements in response to said old haptic elements no longer beingtouched, wherein said set of haptic elements represents said contiguoushaptic elements.
 9. The input/output device of claim 1, furthercomprising: a display, coupled to said plurality of haptic elements,wherein said each one of said plurality of haptic elements correspond toone of a plurality of areas displayed on said display, and said hapticeffect provided by said at least one of said plurality of hapticelements is related to a one of said plurality of areas displayed onsaid display corresponding to said at least one of said plurality ofhaptic elements.
 10. The input/output device of claim 1, wherein said atleast one of said plurality of haptic elements corresponds to one of aplurality of areas displayed on a display; and said haptic effectproduced by said at least one of said plurality of haptic elements isrelated to information displayed in said one of said plurality of areas.11. The input/output device of claim 1, wherein certain ones of saidplurality of haptic elements are configured to produced a hapticresponse at a respective contact surface of said certain ones of saidplurality of haptic elements in response to said respective contactsurface of said certain ones of said plurality of haptic elements beingtouched.
 12. The input/output device of claim 11, wherein each one ofsaid certain ones of said plurality of haptic elements is configured toprovide concurrent input and output when touched concurrently.
 13. Theinput/output device of claim 11, wherein said certain ones of saidplurality of haptic elements are configured to provide input from andoutput to concurrent touches in a plurality of locations.
 14. Theinput/output device of claim 11, wherein each one of said certain onesof said plurality of haptic elements further comprise an actuatorcoupled to a corresponding contact surface and configured to provideforce feedback to said corresponding contact surface.
 15. Theinput/output device of claim 14, wherein said actuator comprises alinear electromagnetic actuator, wherein said contact surface is coupledto said linear electromagnetic actuator.
 16. The input/output device ofclaim 15, wherein said actuator further comprises a spring, wherein saidspring is coupled to said contact surface to maintain said actuator inan extended position when said contact surface is not being touched. 17.The input/output device of claim 14, wherein said each one of saidplurality of haptic elements further comprises a force sensor coupledbetween said contact surface and said actuator.
 18. The input/outputdevice of claim 17, wherein said force sensor is a piezoelectric forcesensor.
 19. The input/output device of claim 11, wherein said certainones of said plurality of haptic elements are further configured toproduce a given haptic effect in a cooperative manner.
 20. Theinput/output device of claim 1, wherein said input/output device isincluded in a computer system.
 21. An input/output device comprising: aplurality of haptic elements, wherein each one of said haptic elementscomprises a contact surface, said contact surfaces define a surface, andeach haptic element of said haptic elements is configured to provide ahaptic effect at the contact surface of said each haptic element uponsaid contact surface of said each haptic element being touched; aplurality of sensors, wherein each one of said sensors includes saidcontact surface of a corresponding one of said haptic elements, and eachone of said sensors is configured to generate information when saidcontact surface included in said corresponding one of said hapticelements is touched; and a processor, coupled to said sensors and saidhaptic elements.
 22. The input/output device of claim 21, wherein asensor of said sensors is further configured to generate informationbased upon a proximity of a pointing device to said contact surface. 23.The input/output device of claim 21, wherein a sensor of said sensors isa resistive film sensor.
 24. The input/output device of claim 23,wherein said resistive film sensor is a four wire resistive film touchsensor.
 25. The input/output device of claim 21, wherein said processoris configured to control a haptic effect produced by certain ones ofsaid haptic elements based on information received from ones of saidsensors corresponding to said certain ones of said haptic elements, andsaid certain ones of said haptic elements are ones of said hapticelements being touched.
 26. The input/output device of claim 25, whereinsaid processor is further configured to cause contiguous haptic elementsof said certain ones of said plurality of haptic elements tocollectively produce the haptic effect.
 27. The input/output device ofclaim 25, wherein said processor is configured to cause contiguoushaptic elements of said certain ones of said plurality of hapticelements to collectively produce the haptic effect by virtue of beingconfigured to calculate a weighted average of a force experienced byeach one of said contiguous haptic elements.
 28. The input/output deviceof claim 25, wherein new haptic elements of said plurality of hapticelements are added to a set of haptic elements in response to said newhaptic elements also being touched and old haptic elements in said setof haptic elements are removed from said set of haptic elements inresponse to said old haptic elements no longer being touched, whereinsaid set of haptic elements represents said contiguous haptic elements.29. The input/output device of claim 21, further comprising: a display,coupled to said plurality of haptic elements, wherein said each one ofsaid plurality of haptic elements correspond to one of a plurality ofareas displayed on said display; and said haptic effect provided by atleast one of said plurality of haptic elements is related to a one ofsaid plurality of areas displayed on said display corresponding to saidat least one of said plurality of haptic elements.
 30. The input/outputdevice of claim 21, wherein certain ones of said plurality of hapticelements are configured to produced a haptic response at the respectivecontact surface included in said certain ones of said plurality ofhaptic elements in response to said respective contact surface of saidcertain ones of said plurality of haptic elements being touched.
 31. Theinput/output device of claim 30, wherein each one of said certain onesof said plurality of haptic elements is configured to provide concurrentinput and output when touched concurrently.
 32. The input/output deviceof claim 30, wherein said certain ones of said plurality of hapticelements are configured to provide input from and output to concurrenttouches in a plurality of locations.
 33. The input/output device ofclaim 30, wherein each one of said certain ones of said plurality ofhaptic elements further comprise an actuator coupled to a correspondingcontact surface of the actuator and configured to provide force feedbackto said corresponding contact surface of the actuator.
 34. Theinput/output device of claim 33, wherein said actuator comprises alinear electromagnetic actuator, wherein said contact surface of theactuator is coupled to said linear electromagnetic actuator.
 35. Theinput/output device of claim 34, wherein said actuator further comprisesa spring, wherein said spring is coupled to said contact surface of theactuator to maintain said actuator in an extended position when saidcontact surface of the actuator is not being touched.
 36. Theinput/output device of claim 33, wherein said each one of said pluralityof haptic elements further comprises a force sensor coupled between saidcontact surface of the actuator and said actuator.
 37. The input/outputdevice of claim 36, wherein said force sensor is a piezoelectric forcesensor.
 38. The input/output device of claim 30, wherein said certainones of said plurality of haptic elements are further configured toproduce a given haptic effect in a cooperative manner.
 39. Theinput/output device of claim 30, wherein said certain ones of saidplurality of haptic elements produce a given haptic effect in acooperative manner by summing forces on ones of said certain ones ofsaid plurality of haptic elements being touched, taking a weightedaverage of positions of said ones of said certain ones of said pluralityof haptic elements being touched, and combining this information inorder to determine a desired feedback force for each one of said ones ofsaid certain ones of said plurality of haptic elements being touched.40. The input/output device of claim 21, wherein said input/outputdevice is included in a computer system.
 41. An input/output devicecomprising: a plurality of haptic elements, wherein each one of saidplurality of haptic elements comprises a contact surface, said contactsurfaces define a surface, and each haptic element of said plurality ofhaptic elements is configured to provide a haptic effect at the contactsurface of said each haptic element upon said contact surface of saideach haptic element being touched; a plurality of sensors, wherein eachone of said plurality of sensors includes said contact surface of acorresponding one of said plurality of haptic elements, each one of saidplurality of sensors is configured to generate information when saidcontact surface coupled to said corresponding one of said plurality ofhaptic elements is touched, and a sensor of said sensors is selectablefrom a plurality of sensors configured to measure at least one of touch,position and motion; and a processor, coupled to said plurality ofsensors and said plurality of haptic elements.
 42. The input/outputdevice of claim 41, wherein said sensor is further configured togenerate information based upon a proximity of a pointing device to saidcontact surface.
 43. The input/output device of claim 41, wherein saidsensor is a resistive film sensor.
 44. The input/output device of claim43, wherein said resistive film sensor is a four wire resistive filmtouch sensor.
 45. The input/output device of claim 41, wherein saidprocessor is configured to control a haptic effect produced by certainones of said plurality of haptic elements based on information receivedfrom ones of said plurality of sensors corresponding to said certainones of said plurality of haptic elements, and said certain ones of saidplurality of haptic elements are ones of said plurality of hapticelements being touched.
 46. The input/output device of claim 45, whereinsaid processor is further configured to cause contiguous haptic elementsof said certain ones of said plurality of haptic elements tocollectively produce the haptic effect.
 47. The input/output device ofclaim 46, wherein new haptic elements of said plurality of hapticelements are added to a set of haptic elements in response to said newhaptic elements also being touched and old haptic elements in said setof haptic elements are removed from said set of haptic elements inresponse to said old haptic elements no longer being touched, whereinsaid set of haptic elements represents said contiguous haptic elements.48. The input/output device of claim 45, wherein said processor isconfigured to cause contiguous haptic elements of said certain ones ofsaid plurality of haptic elements to collectively produce the hapticeffect by virtue of being configured to calculate a weighted average ofa force experienced by each one of said contiguous haptic elements. 49.The input/output device of claim 41, further comprising: a display,coupled to said plurality of haptic elements, wherein said each one ofsaid plurality of haptic elements correspond to one of a plurality ofareas displayed on said display, and said haptic effect provided by atleast one of said plurality of haptic elements is related to a one ofsaid plurality of areas displayed on said display corresponding to saidat least one of said plurality of haptic elements.
 50. The input/outputdevice of claim 41, wherein certain ones of said plurality of hapticelements are configured to produced a haptic response at the respectivecontact surface coupled to said certain ones of said plurality of hapticelements in response to said respective contact surface of said certainones of said plurality of haptic elements being touched.
 51. Theinput/output device of claim 50, wherein each one of said certain onesof said plurality of haptic elements is configured to provide concurrentinput and output when touched concurrently.
 52. The input/output deviceof claim 50, wherein said certain ones of said plurality of hapticelements are configured to provide input from and output to concurrenttouches in a plurality of locations.
 53. The input/output device ofclaim 50, wherein each one of said certain ones of said plurality ofhaptic elements further comprise an actuator coupled to a correspondingcontact surface of the actuator and configured to provide force feedbackto said corresponding contact surface of the actuator.
 54. Theinput/output device of claim 53, wherein said actuator comprises alinear electromagnetic actuator, wherein said contact surface of theactuator is coupled to said linear electromagnetic actuator.
 55. Theinput/output device of claim 54, wherein said actuator further comprisesa spring, wherein said spring is coupled to said contact surface of theactuator to maintain said actuator in an extended position when saidcontact surface of the actuator is not being touched.
 56. Theinput/output device of claim 53, wherein said each one of said pluralityof haptic elements further comprises a force sensor coupled between saidcontact surface of the actuator and said actuator.
 57. The input/outputdevice of claim 56, wherein said force sensor is a piezoelectric forcesensor.
 58. The input/output device of claim 50, wherein said certainones of said plurality of haptic elements are further configured toproduce a given haptic effect in a cooperative manner.
 59. Theinput/output device of claim 50, wherein said certain ones of saidplurality of haptic elements produce a given haptic effect in acooperative manner by summing forces on ones of said certain ones ofsaid plurality of haptic elements being touched, taking a weightedaverage of positions of said ones of said certain ones of said pluralityof haptic elements being touched, and combining this information inorder to determine a desired feedback force for each one of said ones ofsaid certain ones of said plurality of haptic elements being touched.60. The input/output device of claim 41, wherein said input/outputdevice is included in a computer system.
 61. A method of communicatinginformation to and from an input/output device having a plurality ofhaptic elements, the method comprising: reading information from asensor coupled to a contact surface of at least one of the plurality ofhaptic elements, wherein each one of said plurality of haptic elementscomprises a contact surface, said contact surfaces define a surface,said sensor is selectable from a plurality of sensors configured tomeasure at least one of touch, position and motion, and said informationrepresents measurement of said at least one of touch, position andmotion; causing said at least one of said plurality of haptic elementsto produce a haptic effect in response to reading the information,wherein each haptic element of said plurality of haptic elements isconfigured to produce the haptic effect at a contact surface of saidhaptic element in response to said contact surface of said hapticelement being touched and each contact surface has said sensor coupledthereto; sensing the touch on a contact surface of certain ones of saidplurality of haptic elements using corresponding ones of said sensors;generating information corresponding to said certain ones of saidplurality of haptic elements based on said touch; and controlling saidhaptic effect provided by said certain ones of said plurality of hapticelements based on said information.
 62. The method of claim 61, furthercomprising: displaying information in a plurality of areas displayed ona display, wherein each one of said plurality of haptic elementscorresponds to one of said plurality of areas.
 63. The method of claim62, wherein said each one of said plurality of haptic elements furthercomprises an actuator coupled to said contact surface of said each oneof said plurality of haptic elements, the method further comprising:applying force to said contact surface of said each one of saidplurality of haptic elements, using said actuator of said each one ofsaid plurality of haptic elements, based on information displayed in acorresponding one of said plurality of areas displayed on said display.64. The method of claim 61, further comprising: sensing the touch onsaid contact surface of said at least one of said plurality of hapticelements using a touch sensor coupled to said contact surface of said atleast one of said plurality of haptic elements; generating informationbased on said contact surface being touched; and controlling said hapticeffect produced by said at least one of said plurality of hapticelements based on information received from said touch sensor.
 65. Themethod of claim 61, further comprising: causing said certain ones ofsaid plurality of haptic elements to produce a coordinated haptic effectregardless of a distribution of force of said touch among said certainones of said plurality of haptic elements.
 66. An input/output devicecomprising: a plurality of haptic elements, wherein each one of saidplurality of haptic elements comprises an input, the input beinggenerated by a sensor included in each one of said plurality of hapticelements, the sensor being coupled to a contact surface of said hapticelement, and an output, the output being a feedback transferred to thecontact surface in response to the input, and each haptic element ofsaid plurality of haptic elements is configured to provide a hapticeffect at a contact surface of said each haptic element upon saidcontact surface of said each haptic element being touched; a pluralityof sensors, wherein said sensor is a one of said plurality of sensors,each one of said plurality of sensors is coupled to a contact surface ofa corresponding one of said plurality of haptic elements, and each oneof said plurality of sensors is configured to generate information uponsaid contact surface of said corresponding one of said plurality ofhaptic elements being touched; and a processor is coupled to saidplurality of sensors and said plurality of haptic elements.
 67. Theinput/output device of claim 66, wherein: said sensor is selectable froma plurality of sensors configured to measure at least one of touch,position and motion.
 68. The input/output device of claim 66, whereinsaid sensor is further configured to generate information based upon aproximity of a pointing device to said contact surface.
 69. Theinput/output device of claim 66, wherein said sensor is a resistive filmsensor.
 70. The input/output device of claim 66, wherein said sensor isa capacitive proximity sensor.
 71. The input/output device of claim 66,wherein said processor is configured to control a haptic effect producedby certain ones of said plurality of haptic elements based oninformation received from ones of said plurality of sensorscorresponding to said certain ones of said plurality of haptic elements,and said certain ones of said plurality of haptic elements are ones ofsaid plurality of haptic elements being touched.
 72. The input/outputdevice of claim 71, wherein said processor is further configured tocause contiguous haptic elements of said certain ones of said pluralityof haptic elements to collectively produce the haptic effect.
 73. Theinput/output device of claim 72, wherein new haptic elements of saidplurality of haptic elements are added to a set of haptic elements inresponse to said new haptic elements also being touched and old hapticelements in said set of haptic elements are removed from said set ofhaptic elements in response to said old haptic elements no longer beingtouched, wherein said set of haptic elements represents said contiguoushaptic elements.
 74. The input/output device of claim 71, wherein saidprocessor is configured to cause contiguous haptic elements of saidcertain ones of said plurality of haptic elements to collectivelyproduce the haptic effect by virtue of being configured to calculate aweighted average of a force experienced by each one of said contiguoushaptic elements.
 75. The input/output device of claim 66, furthercomprising: a display, coupled to said plurality of haptic elements,wherein said each one of said plurality of haptic elements correspond toone of a plurality of areas displayed on said display, and said hapticeffect provided by at least one of said plurality of haptic elements isrelated to a one of said plurality of areas displayed on said displaycorresponding to said at least one of said plurality of haptic elements.76. The input/output device of claim 66, wherein certain ones of saidplurality of haptic elements are configured to produced a hapticresponse at a respective contact surface of said certain ones of saidplurality of haptic elements in response to said respective contactsurface of said certain ones of said plurality of haptic elements beingtouched.
 77. The input/output device of claim 76, wherein each one ofsaid certain ones of said plurality of haptic elements is configured toprovide concurrent input and output when touched concurrently.
 78. Theinput/output device of claim 76, wherein said certain ones of saidplurality of haptic elements are configured to provide input from andoutput to concurrent touches in a plurality of locations.
 79. Theinput/output device of claim 76, wherein each one of said certain onesof said plurality of haptic elements further comprise an actuatorcoupled to a corresponding contact surface and configured to provideforce feedback to said corresponding contact surface.
 80. Theinput/output device of claim 79, wherein said actuator comprises alinear electromagnetic actuator, wherein said contact surface is coupledto said linear electromagnetic actuator.
 81. The input/output device ofclaim 79, wherein said actuator further comprises a spring, wherein saidspring is coupled to said contact surface to maintain said actuator inan extended position when said contact surface is not being touched. 82.The input/output device of claim 79, wherein said each one of saidplurality of haptic elements further comprises a force sensor coupledbetween said contact surface and said actuator.
 83. The input/outputdevice of claim 82, wherein said force sensor is a piezoelectric forcesensor.
 84. The input/output device of claim 76, wherein said certainones of said plurality of haptic elements produce a given haptic effectin a cooperative manner by summing forces on ones of said certain onesof said plurality of haptic elements being touched, taking a weightedaverage of positions of said ones of said certain ones of said pluralityof haptic elements being touched, and combining this information inorder to determine a desired feedback force for each one of said ones ofsaid certain ones of said plurality of haptic elements being touched.85. The input/output device of claim 66, wherein said certain ones ofsaid plurality of haptic elements are further configured to produce agiven haptic effect in a cooperative manner.
 86. The input/output deviceof claim 66, wherein said input/output device is included in a computersystem.
 87. A method of operating a plurality of haptic elements, themethod comprising: touching a contact surface of at least one of theplurality of haptic elements, each of the plurality of haptic elementscomprising an input and an output; sensing the touch by a sensor togenerate the input to said at least one of the plurality of hapticelements; preparing a feedback in response to the input; transferringthe feedback to the contact surface as the output, wherein each hapticelement of said plurality of haptic elements is configured to producethe haptic effect at a contact surface of said haptic element inresponse to said contact surface of said haptic element being touchedand each contact surface has said sensor coupled thereto; sensing thetouch on a contact surface of certain ones of said plurality of hapticelements using corresponding ones of said sensors; generatinginformation corresponding to said certain ones of said plurality ofhaptic elements based on said touch; and preparing said feedback foreach of said certain ones of said plurality of haptic elements based onsaid information.
 88. The method of claim 87, further comprising:displaying information represented by the input in a plurality of areasdisplayed on a display, wherein each one of said plurality of hapticelements corresponds to one of said plurality of areas.
 89. The methodof claim 87, wherein said at least one of said plurality of hapticelements further comprises an actuator coupled to said contact surface,the method further comprising: applying force representing the output tosaid contact surface of said at least one of said plurality of hapticelements, using said actuator of said at least one of said plurality ofhaptic elements, based on information displayed in a corresponding oneof said plurality of areas displayed on said display.
 90. The method ofclaim 87, preparing the feedback further comprises: reading informationrepresenting the input; generating the information based on said contactsurface being touched; and controlling the feedback prepared based onthe information received from said touch sensor.
 91. The method of claim87, further comprising: causing said certain ones of said plurality ofhaptic elements to produce a coordinated haptic effect regardless of adistribution of force of said touch among said certain ones of saidplurality of haptic elements.
 92. An input/output device comprising: aplurality of haptic elements, wherein each one of said plurality ofhaptic elements comprises a contact surface, said contact surfacesdefine a surface, and at least one of said plurality of haptic elementsis configured to produce a haptic effect at a contact surface of said atleast one of said plurality of haptic elements; a sensor, wherein saidsensor is coupled to said contact surface of said at least one of saidplurality of haptic elements, said sensor is configured to generateinformation in response to said contact surface being touched, and saidat least one of said plurality of haptic elements is configured toproduce said haptic effect based on said information; and a processor,coupled to said sensor and said at least one of said plurality of hapticelements, wherein said processor is configured to control said hapticeffect produced by said at least one of said plurality of hapticelements based on information received from said sensor, certain ones ofsaid plurality of haptic elements are configured to produced a hapticresponse at a respective contact surface of said certain ones of saidplurality of haptic elements in response to said respective contactsurface of said certain ones of said plurality of haptic elements beingtouched, and said certain ones of said plurality of haptic elementsproduce a given haptic effect in a cooperative manner by summing forceson ones of said certain ones of said plurality of haptic elements beingtouched, taking a weighted average of positions of said ones of saidcertain ones of said plurality of haptic elements being touched, andcombining this information in order to determine a desired feedbackforce for each one of said ones of said certain ones of said pluralityof haptic elements being touched.
 93. An input/output device comprising:a plurality of haptic elements, wherein each one of said plurality ofhaptic elements comprises a contact surface, said contact surfacesdefine a surface, and at least one of said plurality of haptic elementsis configured to produce a haptic effect at a contact surface of said atleast one of said plurality of haptic elements; and a sensor, whereinsaid sensor includes said contact surface of said at least one of saidplurality of haptic elements, said sensor is configured to generateinformation in response to said contact surface being touched, said atleast one of said plurality of haptic elements is configured to producesaid haptic effect based on said information, said at least one of saidplurality of haptic elements corresponds to one of a plurality of areasdisplayed on a display, and said haptic effect produced by said at leastone of said plurality of haptic elements is related to informationdisplayed in said one of said plurality of areas.
 94. An input/outputdevice comprising: a plurality of haptic elements, wherein each one ofsaid plurality of haptic elements comprises a contact surface, saidcontact surfaces define a surface, and at least one of said plurality ofhaptic elements is configured to produce a haptic effect at a contactsurface of said at least one of said plurality of haptic elements; and asensor, wherein said sensor is coupled to said contact surface of saidat least one of said plurality of haptic elements, said sensor isselectable from a plurality of sensors configured to measure at leastone of touch, position and motion, said sensor is configured to generateinformation in response to said contact surface being touched, said atleast one of said plurality of haptic elements is configured to producesaid haptic effect based on said information, said at least one of saidplurality of haptic elements corresponds to one of a plurality of areasdisplayed on a display, and said haptic effect produced by said at leastone of said plurality of haptic elements is related to informationdisplayed in said one of said plurality of areas.
 95. An input/outputdevice comprising: a plurality of haptic elements, wherein each one ofsaid plurality of haptic elements comprises: an input, the input beinggenerated by a sensor included in each one of said plurality of hapticelements, the sensor being coupled to a contact surface of said hapticelement; an output, the output being a feedback transferred to thecontact surface in response to the input, said at least one of saidplurality of haptic elements corresponds to one of a plurality of areasdisplayed on a display, and said haptic effect produced by said at leastone of said plurality of haptic elements is related to informationdisplayed in said one of said plurality of areas.